02563 Generative Methods for Comptuer Graphics - Spring 2025

The purpose of this exercise is to familiarize yourself with neural shape representation. This is a relatively new research field that has gained significant interest over the past ~6 years. We will be examining one of the most influential papers in this field, namely DeepSDF, which introduces the Neural Implicit Surface Representation. In this paper, the authors approximate the Signed Distance Field (SDF) for various shapes using a simple neural network called a Multilayer Perceptron (MLP). This approach allows them to generate 3D shapes, interpolate smoothly between different shapes, and auto-complete partial shapes.

In this exercise, you will first learn how to extract a discrete triangle mesh from a pretrained MLP that approximates the SDF of a 3D shape. Next, you will explore how to interpolate between two different 3D shapes using a pretrained MLP that has been trained to approximate the SDF of both shapes. Finally, you will train your own MLP to approximate the SDF of a different set of two shapes and evaluate its performance in interpolating between them. To accomplish this, you must complete the dataloader, which generates the training data for the network, and then train the network. Lastly, you will answer a few questions regarding Neural Implicit Surface Representations.

This notebooks builds on the following papers:

1. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation by Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove
   
   
2. Learning Smooth Neural Functions via Lipschitz Regularization by Hsueh-Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, Or Litany

This notebook is meant to be run on Google Colab, so you can utilize their GPUs. However, it should also work, if you want to run it locally. Just make sure that you have PyTorch and PyGEL installed. It should also be possible to run this notebook without access to a GPU - it will just be a little bit slower.

If you are not familiar with PyGEL, take a look at this introduction and the reference documentation. Note especially the m.triangulate_face(f,mode='v') function.

The bunny and Spot were used courtesy of the Stanford 3D scanning repository and Professor Keenan Crane.

Setup Initial Configurations

Here you install the needed Python packages as well as mouting your Google drive, so you can access the content in the directory of the notebook from the notebook. If you run this notebook on your local machine, you don't have to do this.

# Install the right packages - You need to run this cell, if you run this notebook on Google Colab
!apt-get install libglu1 libgl1 &> /dev/null
!pip install PyGEL3D &> /dev/null
!pip3 uninstall --yes torch torchaudio torchvision torchtext torchdata &> /dev/null
!pip3 install torch torchaudio torchvision torchtext torchdata &> /dev/null
!pip3 install plotly==5.24.1

# # Mount (Connect) your google home drive - Essentially, allow Google to find and retrieve the files
# from google.colab import drive
# drive.mount('/content/drive')
# If you want to check that you mounted the drive correctly, you can use the command below to see what is in your drive
#!ls drive/'My Drive'

# Set directory
drive_path = '/home/ubuntu/study/SIV/Gen/Generative-methods/week5/Ex_Neural_Implicit_Surfaces'

import sys
sys.path.append(drive_path)

# Import ptyhon packages and helper functions from the "utils" python file
from utils import *

# Load data
m_spot = hmesh.obj_load(os.path.join(drive_path, "spot.obj"))
m_bunny = hmesh.obj_load(os.path.join(drive_path,"bunny.obj"))
m_wolf = hmesh.obj_load(os.path.join(drive_path,"wolf.obj"))

# Select CPU or GPU
# If device is cpu, then go to menu and select Runtime -> Change runtime type -> GPU and restart the notebook by going to Runtime -> Restart session.
# If it prints cuda:0, then you have access to a GPU
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print("Using device:", device)

Using device: cuda:0

Display one of the meshes (Spot the cow)

display_meshes(m_spot)

Set up the hyper parameters for the network. You can change these, if you like

latent_vector_size = 256
no_sampled_bounding_box_points = 1000
no_sampled_surface_points = 2000
mesh_resolution = 80
network_hidden_layers = [512,512,512,512,512,512,512,512]
network_leanring_rate = 0.0001
latent_vector_learning_rate = 0.001
num_interpolations = 5
num_epochs = 10000
lipschitz_cosntant = 0.000001
pretrained_model = os.path.join(drive_path,"pretrained_model.pt") # A pretrained model, which approximates the SDF of spot and the wolf.
your_best_model = os.path.join(drive_path,"your_best_model.pt") # When you train the network, the model is saved as "your_best_model.pt"

Part 1: Extract the mesh from pretrained network

In this part of the exercise, you are given a pretrained network that has been trained on Spot and the Wolf. Your task is to extract a triangle mesh from this learned representation

# Network
net = Network(input_size=latent_vector_size + 3, hidden_layers=network_hidden_layers, device=device)
net = net.to(device)
# Latent vectors
lv = Latent_vectors(0.0, 0.01, latent_vector_size, 2, device)

# Load the pretrained model
model = pretrained_model # Use the pretrained model
if os.path.exists(model):
    checkpoint = torch.load(model,map_location=device)
    net.load_state_dict(checkpoint['net_state_dict'])
    net.normalize_params()
    net.eval()
    latent_vectors = checkpoint['latent_vectors']
else:
    print("Error: The pretrained model does not exist")

def inference(net : Network, latent_vector : torch.Tensor, mesh_resolution : int, device : torch.device) -> hmesh.Manifold:
    net.normalize_params()
    net.eval()

    # NB: When inputting the latent vector and 3D point into the network,
    # the order should be: [3D_point,latent_vector]
    x = np.linspace(-1, 1, mesh_resolution)
    y = np.linspace(-1, 1, mesh_resolution)
    z = np.linspace(-1, 1, mesh_resolution)

    # Create a meshgrid
    X, Y, Z = np.meshgrid(x, y, z, indexing='ij')

    # Flatten the grid to get a list of points
    points = np.vstack([X.ravel(), Y.ravel(), Z.ravel()]).T

    points = torch.from_numpy(points).to(device)

    network_input = torch.hstack((points, latent_vector.repeat(len(points),1))).float()

    grid_SDF = net.eval_forward(network_input).reshape(mesh_resolution, mesh_resolution, mesh_resolution)

    grid_SDF = grid_SDF.cpu().detach().numpy()

    # Function that extracts an mesh "m_recovered" from a point grid
    # Input is a point grid of SDF values called grid_SDF
    m_recovered = hmesh.volumetric_isocontour(grid_SDF, make_triangles=True, high_is_inside=False)

    # Function that maps the positions of the point grid to the original space
    # dims is the dimensions of your point grid
    xform = XForm(-1.0, 1.0, (mesh_resolution, mesh_resolution, mesh_resolution))

    pos = m_recovered.positions()
    for v in m_recovered.vertices():
        pos[v] = xform.map(pos[v])

    return m_recovered

mesh = m_spot # Select a mesh
latent_vector = latent_vectors[0] # Find the latent vector belonging to that mesh

m_recovered = inference(net, latent_vector, mesh_resolution, device)

# Save the recovered mesh
hmesh.obj_save(os.path.join(drive_path,"m_recovered.obj"),m_recovered)

# Display the ground truth mesh in red and the reconstructed mesh in blue
if (m_recovered.no_allocated_vertices() != 0):
    display_meshes(m_recovered, mesh) # If you want to display both the reconstructed mesh and the original mesh
    # display_meshes(m_recovered) # If you only want to display the reconstructed mesh

Part 2: Interpolate between the shapes

Use your method that extracts a triangle mesh from the learned signed distance field to interpolate between the two shapes using the latent vectors for the shapes

interpolated_meshes = [] # A list that appends the interpolated meshes

t = np.linspace(0,1,num_interpolations)

for ii in range(num_interpolations):

    interpolated_latent_vector = (1.0 - t[ii]) * latent_vectors[0] + t[ii] * latent_vectors[1] # The interpolated latent vector

    interpolated_mesh = inference(net, interpolated_latent_vector, mesh_resolution, device)
    hmesh.obj_save(os.path.join(drive_path,"m_interpolated_" + str(ii) + ".obj"),interpolated_mesh)
    interpolated_meshes.append( interpolated_mesh )

# If you run this notebook locally on your own machine, you can run the code in this cell.
# viewer = gl.Viewer()
# for i in range(num_interpolations):
#    viewer.display(interpolated_meshes[i],mode='w')
# del viewer

noisy_lattent = lv.latent_vectors[0] + latent_vectors[0] # Add noise to the latent vector
noisy_mesh = inference(net, noisy_lattent, mesh_resolution, device)
display_meshes(m_spot,noisy_mesh)

# Display your approximation of the bunny as a mesh

m_bunny_approximated = interpolated_meshes[0]
display_meshes(m_bunny, m_bunny_approximated)

# Display your approximation of the Wolf as a mesh

m_wolf_approximated = interpolated_meshes[-1]
display_meshes(m_wolf, m_wolf_approximated)

# Display your approximation of the shape that is half bunny, half wolf

m_half_bunny_wolf_approximated = interpolated_meshes[math.floor(len(interpolated_meshes)/2)]
display_meshes(m_half_bunny_wolf_approximated)

Part 3: The actual exercise. Generate the data to train your own network

In this part of the exercise you are supposed to complete the code for the dataloader below, so you can generate the data, which is need to train the network in Part 4 of the exercise

# ---------------------------------------------------------------------------- #
# Class for handling sampling data for ssdf
#
# mesh_list             list - A list of PyGEL3D meshes (the meshes that you want the network to approximate the SDF of)
# no_surface_points     int - The number of points to be sampled near the surface of the shape
# no_box_points         int - The number of points to be sampled in the bounding box of the shape
# mu                    float - The mean of the multivariate normal distribution, from which we sample a normal vector used to offset the sampled points on the surface
# sigma                 float - The standard deviation of the multivariate normal distribution, from which we sample a normal vector used to offset the sampled points on the surface
# ---------------------------------------------------------------------------- #
def sample_random_point_on_triangle(mesh: hmesh, triangle: hmesh) -> tuple:
    """
    Sample a random point on a triangle and return the point and the triangle's normal.
    """
    pos = mesh.positions()
    vertex_indices = np.array(mesh.circulate_face(triangle, mode='v'))
    positions_3D = pos[vertex_indices]
    
    # Compute the normal using the first three vertices of the triangle
    edge1 = positions_3D[1] - positions_3D[0]
    edge2 = positions_3D[2] - positions_3D[0]
    normal = np.cross(edge1, edge2)
    norm_length = np.linalg.norm(normal)
    if norm_length != 0:
        normal = normal / norm_length
    
    barycentric = np.random.rand(3)
    barycentric /= barycentric.sum()
    point = np.dot(barycentric, positions_3D)
    return point, normal


class meshData(Dataset):
    """Mesh dataset."""

    def __init__(self, mesh_list : list, no_surface_points : int, no_box_points : int, mu : float, sigma : float) -> None:

        self.mesh_list = mesh_list
        self.mesh_distances = []
        for mesh in mesh_list:
            self.mesh_distances.append(hmesh.MeshDistance(mesh))

        # YOUR CODE HERE
        #----------------------------------------
        self.triangle_areas = [np.array([mesh.area(face) for face in mesh.faces()]) for mesh in mesh_list]
        self.triangle_areas = [area/area.sum() for area in self.triangle_areas]

        self.num_surface_points = no_surface_points
        self.num_box_points = no_box_points
        self.mu_normal_vector = mu
        self.sigma_normal_vector = sigma

    def __len__(self):
        return len(self.mesh_list)

    def __getitem__(self, idx):
        # So if you provide the index as a tensor, which you would do, when you sample,
        # then this is converted to a list.
        if torch.is_tensor(idx):
            idx = idx.tolist()

        # Find the mesh and signed distance field by using the idx.
        mesh = self.mesh_list[idx]
        mesh_dist = self.mesh_distances[idx]

        # YOUR CODE HERE
        #----------------------------------------
        # Sample the points close to the surface of the mesh and in the box from [-1,-1,-1] to [1,1,1], which encloses the surface

        # ------------------------------------------
        # 3D points
        # ------------------------------------------
        points3d = []
        
        ## sample points close to the surface
        for i in range(self.num_surface_points//2):
            # Sample a triangle
            triangle = np.random.choice(mesh.faces(), p=self.triangle_areas[idx])
            # Sample a point on the triangle
            point, normal = sample_random_point_on_triangle(mesh, triangle)
            # Add a bit of noise in the face normal direction
            noise = np.random.normal(self.mu_normal_vector, self.sigma_normal_vector, 3)
            point_outsied = point + noise * normal
            point_inside = point - noise * normal
            points3d.append(point_outsied)
            points3d.append(point_inside)
        
        # ## sample points in the bounding box
        for _ in range(self.num_box_points):
            point = np.random.uniform(-1, 1, 3)
            points3d.append(point)
        

        points3d = np.array(points3d)

        # ------------------------------------------
        # sdf - Signed Distance Field
        # ------------------------------------------
        sdf_values = mesh_dist.signed_distance(points3d).reshape((len(points3d),1)) # Compute the SDF of the sampled 3D points
        sdf_values = torch.from_numpy(sdf_values).float() # Convert to PyTorch Tensor
        points3d = torch.from_numpy(points3d) # Convert to PyTorch Tensor

        sample = {'shape_id': idx, 'points3d': points3d, 'sdf_values': sdf_values}

        return sample

# Debugging - Make sure that the point cloud is sampled around the mesh
mesh_list = [m_bunny, m_wolf]
meshDataset = meshData(mesh_list=mesh_list,
                       no_surface_points = no_sampled_surface_points,
                       no_box_points=no_sampled_bounding_box_points,
                       mu=0.0, sigma=0.05)

# Display the points together with the mesh. Is the result as you expected?
display_mesh_and_points(mesh_list[1], meshDataset[1]['points3d'].detach().numpy())

Part 4: Train the network

In this part of the exercise you are supposed to train the network yourself - The code has already been writtne, so you should simply run the next cells of code

mesh_list = [m_bunny, m_wolf]
meshDataset = meshData(mesh_list=mesh_list,
                       no_surface_points = no_sampled_surface_points,
                       no_box_points=no_sampled_bounding_box_points,
                       mu=0.0, sigma=0.05)

dataloader = DataLoader(meshDataset, batch_size=1, shuffle=True, num_workers=0, pin_memory=True)

# Network
net = Network(input_size=latent_vector_size + 3, hidden_layers=network_hidden_layers, device=device)
net = net.to(device)
# Latent vectors
lv = Latent_vectors(0.0, 0.01, latent_vector_size, 2, device)

# Optimizer
net_optimizer = optim.Adam(net.parameters(), lr=network_leanring_rate)
latent_vector_optimizer = optim.Adam([lv.latent_vectors], lr=latent_vector_learning_rate)

# Loss function
L1_loss = nn.L1Loss(reduction='sum')

# -----------------------------
# Note: You should get a quite good neural representation, if you use the hyper parameters above and train for approximiately 1200 epochs.
# -----------------------------

min_loss = math.inf
# To make sure that you do not overwrite your best model, import your model and find the minimum loss.
if os.path.exists(your_best_model):
    checkpoint = torch.load(your_best_model,map_location=device)
    min_loss = checkpoint['loss']
losses = []
iteration = []

for epoch in range(num_epochs):
    current_loss = 0.0

    net.train()
    for i, batch in enumerate(dataloader):
        # Zero the network and latent vector gradients
        net_optimizer.zero_grad()
        latent_vector_optimizer.zero_grad()

        # Get data from dataloader
        mesh_index, batch_points3d, batch_target = batch['shape_id'][0], batch['points3d'][0].to(device), batch['sdf_values'][0].to(device)
        N = batch_points3d.shape[0]
        # Concatenate the batch of 3d points with the latent vector
        net_input = torch.cat((batch_points3d, lv.latent_vectors[mesh_index].repeat(N, 1)), axis=1).float().to(device)
        # Feed the data through the network
        net_output = net.train_forward(net_input)

        # Compute the batch loss
        batch_loss = L1_loss(net_output, batch_target) + lipschitz_cosntant * net.get_lipshitz_loss().to(device)

        # Compute the gradients
        batch_loss.backward()

        # Take a step with the optimizer
        net_optimizer.step()
        latent_vector_optimizer.step()

        current_loss += float(batch_loss)

    #---------------------------------------
    # Save best model as your_best_model.pt
    #---------------------------------------
    if current_loss < min_loss:
        min_loss = current_loss
        torch.save({
            'epoch': epoch,
            'net_state_dict': net.state_dict(),
            'latent_vectors': lv.latent_vectors,
            'net_optimizer_state_dict': net_optimizer.state_dict(),
            'loss': current_loss,
        },your_best_model)

    #----------------------------------
    # Display the training error
    #----------------------------------
    losses.append(current_loss) # All batch losses
    iteration.append(epoch)
    plt.plot(iteration,losses,color='black')
    display.clear_output(wait=True)
    plt.grid()
    display.display(plt.gcf())
    plt.xlabel("Epoch")
    plt.ylabel("Loss")
    plt.title("Training Loss")
    #time.sleep(0.1)
    plt.grid()
    display.clear_output(wait=True)

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[23], line 63
     61 display.clear_output(wait=True)
     62 plt.grid()
---> 63 display.display(plt.gcf())
     64 plt.xlabel("Epoch")
     65 plt.ylabel("Loss")

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/IPython/core/display_functions.py:298, in display(include, exclude, metadata, transient, display_id, raw, clear, *objs, **kwargs)
    296     publish_display_data(data=obj, metadata=metadata, **kwargs)
    297 else:
--> 298     format_dict, md_dict = format(obj, include=include, exclude=exclude)
    299     if not format_dict:
    300         # nothing to display (e.g. _ipython_display_ took over)
    301         continue

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/IPython/core/formatters.py:238, in DisplayFormatter.format(self, obj, include, exclude)
    236 md = None
    237 try:
--> 238     data = formatter(obj)
    239 except:
    240     # FIXME: log the exception
    241     raise

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/decorator.py:232, in decorate.<locals>.fun(*args, **kw)
    230 if not kwsyntax:
    231     args, kw = fix(args, kw, sig)
--> 232 return caller(func, *(extras + args), **kw)

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/IPython/core/formatters.py:282, in catch_format_error(method, self, *args, **kwargs)
    280 """show traceback on failed format call"""
    281 try:
--> 282     r = method(self, *args, **kwargs)
    283 except NotImplementedError:
    284     # don't warn on NotImplementedErrors
    285     return self._check_return(None, args[0])

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/IPython/core/formatters.py:402, in BaseFormatter.__call__(self, obj)
    400     pass
    401 else:
--> 402     return printer(obj)
    403 # Finally look for special method names
    404 method = get_real_method(obj, self.print_method)

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/IPython/core/pylabtools.py:170, in print_figure(fig, fmt, bbox_inches, base64, **kwargs)
    167     from matplotlib.backend_bases import FigureCanvasBase
    168     FigureCanvasBase(fig)
--> 170 fig.canvas.print_figure(bytes_io, **kw)
    171 data = bytes_io.getvalue()
    172 if fmt == 'svg':

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/backend_bases.py:2184, in FigureCanvasBase.print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, pad_inches, bbox_extra_artists, backend, **kwargs)
   2180 try:
   2181     # _get_renderer may change the figure dpi (as vector formats
   2182     # force the figure dpi to 72), so we need to set it again here.
   2183     with cbook._setattr_cm(self.figure, dpi=dpi):
-> 2184         result = print_method(
   2185             filename,
   2186             facecolor=facecolor,
   2187             edgecolor=edgecolor,
   2188             orientation=orientation,
   2189             bbox_inches_restore=_bbox_inches_restore,
   2190             **kwargs)
   2191 finally:
   2192     if bbox_inches and restore_bbox:

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/backend_bases.py:2040, in FigureCanvasBase._switch_canvas_and_return_print_method.<locals>.<lambda>(*args, **kwargs)
   2036     optional_kws = {  # Passed by print_figure for other renderers.
   2037         "dpi", "facecolor", "edgecolor", "orientation",
   2038         "bbox_inches_restore"}
   2039     skip = optional_kws - {*inspect.signature(meth).parameters}
-> 2040     print_method = functools.wraps(meth)(lambda *args, **kwargs: meth(
   2041         *args, **{k: v for k, v in kwargs.items() if k not in skip}))
   2042 else:  # Let third-parties do as they see fit.
   2043     print_method = meth

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/backends/backend_agg.py:481, in FigureCanvasAgg.print_png(self, filename_or_obj, metadata, pil_kwargs)
    434 def print_png(self, filename_or_obj, *, metadata=None, pil_kwargs=None):
    435     """
    436     Write the figure to a PNG file.
    437 
   (...)
    479         *metadata*, including the default 'Software' key.
    480     """
--> 481     self._print_pil(filename_or_obj, "png", pil_kwargs, metadata)

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/backends/backend_agg.py:429, in FigureCanvasAgg._print_pil(self, filename_or_obj, fmt, pil_kwargs, metadata)
    424 def _print_pil(self, filename_or_obj, fmt, pil_kwargs, metadata=None):
    425     """
    426     Draw the canvas, then save it using `.image.imsave` (to which
    427     *pil_kwargs* and *metadata* are forwarded).
    428     """
--> 429     FigureCanvasAgg.draw(self)
    430     mpl.image.imsave(
    431         filename_or_obj, self.buffer_rgba(), format=fmt, origin="upper",
    432         dpi=self.figure.dpi, metadata=metadata, pil_kwargs=pil_kwargs)

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/backends/backend_agg.py:382, in FigureCanvasAgg.draw(self)
    379 # Acquire a lock on the shared font cache.
    380 with (self.toolbar._wait_cursor_for_draw_cm() if self.toolbar
    381       else nullcontext()):
--> 382     self.figure.draw(self.renderer)
    383     # A GUI class may be need to update a window using this draw, so
    384     # don't forget to call the superclass.
    385     super().draw()

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/artist.py:94, in _finalize_rasterization.<locals>.draw_wrapper(artist, renderer, *args, **kwargs)
     92 @wraps(draw)
     93 def draw_wrapper(artist, renderer, *args, **kwargs):
---> 94     result = draw(artist, renderer, *args, **kwargs)
     95     if renderer._rasterizing:
     96         renderer.stop_rasterizing()

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/artist.py:71, in allow_rasterization.<locals>.draw_wrapper(artist, renderer)
     68     if artist.get_agg_filter() is not None:
     69         renderer.start_filter()
---> 71     return draw(artist, renderer)
     72 finally:
     73     if artist.get_agg_filter() is not None:

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/figure.py:3257, in Figure.draw(self, renderer)
   3254             # ValueError can occur when resizing a window.
   3256     self.patch.draw(renderer)
-> 3257     mimage._draw_list_compositing_images(
   3258         renderer, self, artists, self.suppressComposite)
   3260     renderer.close_group('figure')
   3261 finally:

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/image.py:134, in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    132 if not_composite or not has_images:
    133     for a in artists:
--> 134         a.draw(renderer)
    135 else:
    136     # Composite any adjacent images together
    137     image_group = []

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/artist.py:71, in allow_rasterization.<locals>.draw_wrapper(artist, renderer)
     68     if artist.get_agg_filter() is not None:
     69         renderer.start_filter()
---> 71     return draw(artist, renderer)
     72 finally:
     73     if artist.get_agg_filter() is not None:

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/axes/_base.py:3210, in _AxesBase.draw(self, renderer)
   3207 if artists_rasterized:
   3208     _draw_rasterized(self.get_figure(root=True), artists_rasterized, renderer)
-> 3210 mimage._draw_list_compositing_images(
   3211     renderer, self, artists, self.get_figure(root=True).suppressComposite)
   3213 renderer.close_group('axes')
   3214 self.stale = False

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/image.py:134, in _draw_list_compositing_images(renderer, parent, artists, suppress_composite)
    132 if not_composite or not has_images:
    133     for a in artists:
--> 134         a.draw(renderer)
    135 else:
    136     # Composite any adjacent images together
    137     image_group = []

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/artist.py:71, in allow_rasterization.<locals>.draw_wrapper(artist, renderer)
     68     if artist.get_agg_filter() is not None:
     69         renderer.start_filter()
---> 71     return draw(artist, renderer)
     72 finally:
     73     if artist.get_agg_filter() is not None:

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/lines.py:807, in Line2D.draw(self, renderer)
    804         gc.set_foreground(lc_rgba, isRGBA=True)
    806         gc.set_dashes(*self._dash_pattern)
--> 807         renderer.draw_path(gc, tpath, affine.frozen())
    808         gc.restore()
    810 if self._marker and self._markersize > 0:

File ~/miniconda3/envs/gen_comp_graphics/lib/python3.12/site-packages/matplotlib/backends/backend_agg.py:130, in RendererAgg.draw_path(self, gc, path, transform, rgbFace)
    128 else:
    129     try:
--> 130         self._renderer.draw_path(gc, path, transform, rgbFace)
    131     except OverflowError:
    132         cant_chunk = ''

KeyboardInterrupt: 

Part 5: Questions

1. How do we convert the neural implicit representation to a triangle mesh?

Answer: We simply sample a grid of points (voxels). Then we know that there will be a face (surface) in between voxels that have opposite sign values.

2. Why is it only possible to use watertight meshes?

Answer: Feed-forward neural networks are created to approximate smooth functions. Because of that, the approximated SDF is also smooth, which in our setting implies that the surface will be watertight.

3. What are the hyperparameters and what influence do they have?

Answer:

• Number of points to sample -> the more points, the easier it should be for the network to approximate the true SDF.
• Mu and sigma in meshDataset -> controls the noise step that we take from the face when sampling the points near the face.
• Latent vector size -> size of the internal representation of the shape. It should store the most important features of the object. If it's too small, it can be hard to express the details of the object. If it's too big, it can be hard to train the model because of the curse of dimensionality.

4. What does the Lipschitz regularization do?

Answer: It constrains the network to be Lipschitz continuous. This important when we want to interpolate between two shapes. If the network is not Lipschitz continuous, the interpolation can be very unstable.

5. Do the recovered surfaces differ a lot from the true surfaces? If so, why?

Answer: The recovered surfaces follow resonably well the true surfaces. However, the recovered surfaces fails to capture some details. The interesting example of it is the wolf's mouth. The recovered wolf has the mouth closed, while the true wolf has the mouth open. This is probably becouse there weren't enough data point around the wolf's mouth to properly capture the shape of the mouth.

# Load the best model
if os.path.exists(your_best_model):
    checkpoint = torch.load(your_best_model,map_location=device)
    net.load_state_dict(checkpoint['net_state_dict'])
    net.normalize_params()
    net.eval()
    latent_vectors = checkpoint['latent_vectors']

# inference
mesh = m_wolf # Select a mesh
latent_vector = latent_vectors[1] # Find the latent vector belonging to that mesh

m_recovered = inference(net, latent_vector, mesh_resolution, device)

# showe the recovered mesh
display_meshes(m_recovered, mesh)

Non-mandatory assignment

1. Try to not include the Lipschitz term in the loss and see what effect it has on the shape approximation.

2. Try experimenting with the size of the latent vector. Do you get better representations when the latent vector is greater in size?

3. Try to train the network on all three shapes or perhaps your own shapes. How do you interpolate between three shapes?

4. Render images of the meshes that you obtained from the interpolation in e.g. Blender, and turn the images into a GIF/small movie using the python script: images_to_gif.py from Learn.